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Three-dimensional reconstruction of an object from projection photographs 



The invention relates to a device and a method for the three-dimensional 
reconstruction of a moving object in a body volume, wherein the body volume can be subject 
to a cyclical self-movement, for example through heartbeat or breathing. 

The execution of intravascular medical interventions such as for example the 
5 placing of a stent at a stenosis of the coronary vessels usually takes place under X-ray 

fluoroscopy observation. For monitoring the placement of the stent, it would be helpful here 
to be able to visualize it three-dimensionally, as accurately as possible. Such visualization is 
however not readily possible with the known three-dimensional imaging methods of 
computer tomography, since due to the heartbeat and respiration, the stent is subject to 
10 constant movement and deformation. 

In this connection, for the three-dimensional reconstruction of vascular trees, 
. we know of the production during a heartbeat phase of two projection series of the vessels 
filled with contrast medium from different directions (S.-Y.J. Shen, J.D. Carroll: "Dynamic 
reconstruction of 3D coronary arterial trees based on a sequence of biplane angiograms", in 
15 Proceedings of the SPIE, vol. 3034, pp. 358-368 (1997)). On the projection photographs, 

distinctive points such as e.g. bifurcations are segmented, and from two projections from the 
same stage in the photo sequences but from different projection directions, the spatial 
position of these distinctive points is reconstructed. Finally, from all such reconstructed 
positions, the movement of the points and thus of the vascular system during the heartbeat is 
20 determined. The application of the method to the localization of objects such as intervention 
devices is not described. Furthermore, the method requires that the photographic sequences 
are recorded at high rate over the duration of precisely one heartbeat. 

Against this background, it was an object on the invention to provide the 
means for localizing an object such as for example an intervention device in the vascular 
25 system of a patient, wherein the object and/or the surrounding body volume can be subject to 
a cyclical or non-cyclical movement. 

This object is fulfilled by a device with the features of claim 1, by a device 
with the features of claim 4, by a method with the features of claim 9 and by a method with 
the features of claim 10. Advantageous embodiments are contained in the dependent claims. 
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According to a first aspect of the invention, the device in accordance with the 
invention serves the purpose of three-dimensional reconstruction of a moving object in a 
body volume. The object can be in particular a body structure (organ, vessel section, the body 
volume itself etc.) or an intervention device on a catheter, such as e.g. a stent, a balloon or a 
5 guide wire, wherein however the invention is not restricted to medical applications. The 
movement of the object can take place relative to the body volume (e.g. advancement of a 
catheter) and/or together with a movement of the body volume (e.g. through patient 
movements, breathing, heartbeat). 

The device comprises an (electronic) memory which contains a series of two- 
10 dimensional projection photographs of the body volume from different directions. Coupled to 
the memory is a data processing unit which is set up to execute the following steps: 

a) The segmentation of the image of at least one feature point of the object or its 
surroundings in the projection photographs. According to the definition, a "feature point of 
the object or its surroundings" here is a point belonging to the object or its surroundings 

1 5 which can be localized relatively well on the projection photographs. For example, it can be a 
vessel branching or a radio-opaque marking on a guide wire or stent. Furthermore, 
"segmentation" is understood in the usual manner to mean the allocation of an image point to 
an object (here: the feature point). For preference, two or more different feature points are 
segmented and respectively subjected to the subsequent further processing formulated for 

20 "the feature poinf \ 

b) The specification of a spatial reference position for the feature point. The 
"true" spatial position of the feature point (at a particular point in time) cannot be established 
with certainty even from several projection photographs, because of the unknown movement 
of the object. For this reason, a reference position is set more or less randomly here, which is 

25 subsequently taken as the basis for interpretation and alignment of the photographs. 

c) Calculation of transformations of the three-dimensional object space and of 
the two-dimensional projection photographs, after the use of which the projection of the 
(transformed) reference position respectively coincides with the (transformed) image of the 
feature point. Typically, one of these transformations is set as the identity, so that effectively 

30 either a true transformation of the object space of a true transformation of the projection 

photographs takes place. A "true" transformation can for example be a translation, rotation, 
dilation (volume change), and/or - in more complex formulations - an affine transformation. 

In step c) for example the projection photographs can be transformed such that 
the image of the feature point that is segmented in step a) comes to rest at a place on which 
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the reference position defined in step b) is projected. Alternatively, for each projection 
photograph the reference position can be transformed into a new position, whose projection 
then lies on the image of the feature point that was segmented in step a). In both cases, 
transformations are established which describe an alignment of the projection photographs 
5 onto the reference position (or an alignment of the reference position onto the projection 
photographs respectively). 

d) The three-dimensional reconstruction of the object and, if applicable, of the 

surroundings of the object, i.e. in general of a region of interest, from the stored projection 
photographs, using the transformations calculated according to step e). 

10 The device has the advantage that the object, or region of interest in the case of 

the object, can be reconstructed using all the projection photographs from the memory. This 
succeeds since changes in the position and shape of the moving object can be compensated 
with the aid of the transformations. It is of particular advantage in this connection that the 
movement of the object can be of practically any kind. 

1 5 The assumed spatial reference position of a feature point in step b) of the 

method can in principle be set as desired. However, advantageously it is defined such that it . 
lies as close as possible to a real or the most probable current position of the feature point. In 
particular, the reference position can be reconstructed geometrically from two projection 
photographs that were taken when the body volume was in the same type of state, but from 

20 different directions. In this case it can be assumed that during the production of the two 

projection photographs, a feature point lay at approximately the same spatial position, which 
can therefore be reconstructed exactly. With regard to the state of the body volume, this can 
be in particular the heartbeat phase, which is typically recorded with an ECG. 

According to a second aspect of the invention, the invention relates to the 

25 three-dimensional reconstruction of an object in a body volume that is subject to cyclical self- 
movement. The object can in particular be a body structure (organ, vessel section etc.) or an 
intervention device on a catheter such as e.g. a stent, a balloon or a guide wire, wherein 
however the invention is not restricted to medical applications. The cyclical self-movement is 
typically one such as is caused by the heartbeat and/or breathing in a biological body. 

30 The device comprises an (electronic) memory which contains a series of two- 

dimensional projection photographs of the body volume from different directions. For each 
of the projection photographs, the memory also includes the corresponding values of a 
parameter that characterizes the cyclical self-movement of the body volume at the point in 
time when the projection photograph is produced. The parameter can be for example an ECG 
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value. Coupled to the memory is a data processing unit that is set up to execute the following 
steps: 

a) Segmentation of the image of at least one feature point of the object in the 
projection photographs. A "feature point of the object" here is, as already explained, a point 

5 belonging to the object, which can be relatively well localized on the projection photographs. 
Preferably, two or more different feature points of the object are segmented and subjected to 
the subsequent further processing formulated for "the feature point". 

b) Classification of said projection photographs into classes, wherein each of the 
classes collects together all the projection photographs that belong to a particular phase of the 

10 cyclical self-movement that is assigned to the class. 

c) Three-dimensional localization of the feature point for each of the classes 
named above from at least two projection photographs of the corresponding class. Such 
localization is possible in principle since according to the requirements, the projection 
photographs have been produced from different directions. Localization is also practically 

15 possible and sensible since the projection photographs that are used originate from the same 
class and thus the same phase of the cyclical self-movement, and the body volume thus has 
approximately the same shape and position in the projection photographs. 

d) Calculation of three-dimensional transformations, wherein a transformation 
describes (amongst other things) the movement or displacement of the feature point during 

20 the transition from one phase of the cyclical self-movement to another. Determination of such 
transformations is possible since from step c), the spatial position of the feature point in the 
different phases of the cyclical self-movement that correspond to the classes is known. In the 
simplest case, the transformation is a translation, which describes the linear displacement of a 
point. If more than two feature points are observed, the transformation can also be a rotation, 

25 a dilation (volume change), and/or — in more complex formulations - an affine 
transformation. 

e) Three-dimensional reconstruction of the object and, if applicable, the 
surroundings of the object, i.e. in general of a region of interest, from the stored projection 
photographs, using the transformations calculated according to step d). 

30 The device has the advantage that the object, or region of interest in the case of 

the object, can be reconstructed using all the projection photographs from the memory. This 
succeeds since changes in the position and shape of the object due to the cyclical self- 
movement of the body volume can be compensated with the aid of the transformations, since 
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the transformations describe - at least approximately - the effect of the self-movement 
mathematically. 

Preferred designs of the invention, which can be applied for each of the two 
devices according to the first and second aspects of the invention, will be elucidated below, 

5 wherein for the sake of simplicity we shall speak only of "the device". 

Segmentation of the images of the at least one feature point in step a) can, in 
the case of the device, take place fully automatically or semi-automatically. In the latter case, 
the device preferably includes an input unit coupled to the data processing unit, such as a 
keyboard and/or a mouse, via which the user can support the segmentation interactively. 

10 The device furthermore comprises at least one image-producing device, with 

which the series of two-dimensional projection photographs of the body volume that are 
stored in the memory can be produced. This device can be in particular an X-ray apparatus 
for producing X-ray projections and/or an NMR device. 

The device furthermore comprises a sensor device for recording a parameter 

15 that characterizes a (or the named) cyclical self-movement of the body volume in parallel 
with the production of the projection photographs. The sensor device can be in particular an 
electrocardiograph device for recording an electrocardiogram and/or a respiration sensor for 
recording the respiration phase. 

The invention furthermore relates to a method for the three-dimensional 

20 reconstruction of a moving object in a body volume based on a quantity of data which 
contains a series of two-dimensional projection photographs of the body volume from 
different directions, comprising the steps: 

a) Segmentation of the image of at least one feature point of the object or its 

surroundings in the projection photographs; 
25 b) Specification of a spatial reference position for each feature point; 

c) Calculation of transformations of the object space and of the projection 

photographs, after the use of which the projections of the transformed reference points in 
each transformed projection photograph coincide with the images of the feature points each 
time; 

30 d) Three-dimensional reconstruction of the object from the projection 

photographs with the aid of the calculated transformations. 

The invention furthermore relates to a method for the three-dimensional 
reconstruction of an object in a body volume that is subject to a cyclical self-movement, 
based on a quantity of data which contains a series of two-dimensional projection 
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photographs of the body volume from different directions together with the respectively 
corresponding values of a parameter that characterizes the cyclical self-movement, 
comprising the steps: 

a) Segmentation of the image of at least one feature point of the object in the 
projection photographs; * 

b) Classification of the projection photographs into classes which each 
correspond to a given phase of the cyclical self-movement; 

c) Three-dimensional localization of said feature point for each of the said 
classes from at least two projection photographs of this class; 

d) Calculation of three-dimensional transformations which describe the 
movement of the localized feature point between different phases of the cyclical self- 
movement; 

e) Three-dimensional reconstruction of the object from the projection 
. photographs with the aid of the calculated transformations. 

The two methods include in general form the steps that can be executed with 
the devices of the type described above. In respect of the explanation of details, advantages 
and further aspects of the method, we refer to the description given above. 

These and other aspects of the invention are apparent from and will be 
elucidated with reference to the embodiments described hereinafter. 

In the drawings: 

Fig. 1 shows a schematic representation of the projection geometry in the 
production of X-ray pictures of a moving object; 

Fig. 2 shows the effect of transformations calculated for the object space for 
compensating the object movement; 

Fig. 3 shows the effect of transformations calculated for the projection 
photographs for compensating the object movement; 

Figs. 4a, b show a schematic flow diagram for a method for the three- 
dimensional reconstruction of an object from X-ray images from different heartbeat phases. 

First of all, a first embodiment of the invention is elucidated on the basis of 
Figures 1 to 3. In the case of the underlying method, the task is to carry out the three- 
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dimensional reconstruction of a moving object (e.g. of a stent or of the body volume itself) 
with the aid of two-dimensional X-ray projection photographs Ai, A 2 , ... A n , ... An of the 
object which have been taken from different projection directions. 

In this regard, Figures 1 to 3 show, in schematic form, the projection geometry 
5 that is present when a series of X-ray photographs is produced during a sweep of a rotating 
X-ray apparatus. Here, the X-ray pictures Ai, A2, ... A n , ... An are taken starting from 
projection centers Fj, F 2 , ... F n , ... Fn- In the Figures, shown schematically on the projection 
photographs is a vessel section which is depicted differently in each case due to the different 
projection directions and the movement of the body volume that is photographed (through 

10 movement by the patient, heartbeat, breathing etc.). Without the movement, the object of 

interest could be reconstructed exactly three-dimensionally from the projection photographs. 
However, in practice the object moves in an unknown manner, so that for a feature point Q 
that is firmly connected to the object, the movement trajectory Q(t) shown schematically in 
Figure 1 results. As a rule therefore each of the projection photographs Ai, A2, ... A n , ... An 

15 records the feature point Q at a different, unknown point of the trajectory. In order 

nevertheless to obtain as good a reconstruction of the object as possible, the method for 
compensating the movement as explained below is proposed. 

The method is based on tracking at least one feature point, such as for example 
a marking point on a catheter or a branching point of a vessel. For this, besides the 

20 aforementioned feature point Q, preferably one or more further feature points, not shown in 
greater detail in Figures 1 to 3, are used. As per the requirements, the images of a feature 
point that are produced through the X-ray projections can be easily localized on the 
projection photographs. Thus for example in the projection photograph A n , the image Pr n (Q) 
of the feature point Q that is produced with the a** 1 projection Pr n can be segmented fully 

25 automatically or semi-automatically. A similar procedure applies for the other projection 
photographs Ai, A 2 , ... An. 

Also shown in Figure 1 are the projection lines that connect the respective 
projection center F n with the image Pr n (Q) of the feature point Q. If the feature point were not 
to move, then all the projection lines would have to intersect in it, so that its spatial position 

30 could be reconstructed exactly. However, due to the movement of the feature point Q there is 
not such an intersection of all the corresponding projection lines; rather, these merely come 
more or less close to one another in a small volume (dotted in Figure 1). 

In the next step of the method, a reference position Qo is established 
(randomly), on which subsequently all the projection photographs Ai, A 2 , ... A n , ... An are to 
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be aligned. For preference, this reference position Qo is placed as close as possible to an 
actual or probable position of the feature point Q. In the case of the method indicated in 
Figure 1, the reference position Qo is for example placed in the center of the node in which 
the individual projection lines cross. Alternatively, the reference position could also be 
6 0 5 placed in the intersection point of two projection lines (if present) that are known to belong to 
projection photographs from a similar state of movement of the body volume (for example to 
the same heartbeat phase in the case of cardiac images). 

Represented in Figure 2 is a first variant of the movement compensation with 
the aid of the selected reference point Qo. Here, for each of the projection photographs such 

10 as for example for the photograph A n , such a displacement of the reference position Qo in the 
space is determined that the projection Pr n of the new position on the projection photograph 
A n falls precisely into the image of the feature point Q. The aforementioned displacement of 
the reference position Q 0 (optionally together with the displacements of other feature points) 
can now be extended to a three-dimensional transformation S n of the entire object space. This 

15 transformation Sn thus describes geometrically the transition (translation* rotation, dilation 
etc.) of the body volume under observation, from the (fictitious) reference state to the state . 
recorded in the projection photograph A n . Since corresponding transformations Si, Si, ... S N 
can also be calculated for the other projection photographs, the result is that the influence of 
the movement of the body volume on the projection photographs Ai, A2, ... A n , ... An can be 

20 compensated computationally. The body volume can then be reconstructed to a high level of 
accuracy from the movement-compensated X-ray photographs. 

Figure 3 shows a method of movement compensation that is an alternative to 
Figure 2 but is equivalent in terms of the results. By contrast to Figure 2, here it is not three- 
dimensional transformations of the object space that are calculated, by two-dimensional 

25 transformations ai, 02, »- cy n » ... of the projection photographs Ai, A 2 , ... A n , ... An. In this 
case, first of all the set reference position Qo is projected computationally onto the respective 
projection photographs Ai, A2, ... A n , ... An with the aid of the known geometric projections 
Pr n , which in the projection photograph An for example leads to the image Pr n (Qo) of the 
reference position. This calculated image does not as a rule coincide with the represented 

30 image Pr n (Q) of the actual feature point. With the aid of the information available (optionally 
also including further feature points) however a two-dimensional transformation a n can be 
defined; when this is used, the projection photograph A n is altered (displaced, rotated etc.) 
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just so that the transformed image of the feature point accords with the calculated image of 
the reference position Qo, i.e. a n (Pr n (Q)) = Pr n (Q). 

Through the calculation and application of all two-dimensional 
transformations <7i, (72, ... <r n > <*n to the projection photographs Ai, A2, ... A n , ... An, as a 
5 result it is ensured that all projection photographs are compensated on a fictitious, static 
spatial position Qo of the feature point This makes it possible for the body volume, or an 
1 object located within it, subsequently to be reconstructed three-dimensionally, using all the 
(transformed) projection photographs. 

As a rule, the randomness in setting the reference positions in the case of the 
10 above method leads to the fact that the three-dimensional reconstruction of the body volume 
can be altered (displaced, rotated, compressed etc.) in relation to the real body volume. For 
many applications, such as for example the positioning of a stent in a vessel, this 
circumstance is hardly a disadvantage in practice. Far more important here is the very sharp, 
hardly blurred representation of the reconstructed volume, which is made possible through 
15 the movement compensation. 1 ■ 

Figure 4 shows a second embodiment of the invention. In the application case 
taken as an example here, this involves the localization of a stent 5 which is fitted to the tip of 
a catheter and is to be unfurled as precisely as possible in the area of a stenosis requiring 
treatment. For observation of the intervention, a rotary X-ray apparatus 1 is available, with 
20 which in one sweep two-dimensional projection photographs A n of the relevant body volume 
can be taken from different projection directions. 

Due to the heartbeat, the stent 5 that is located in the coronary vessels is 
constantly moved and possibly also deformed (similar considerations apply for breathing, 
which for the sake of simplifying the representation is however omitted in the following; 
25 fundamentally it can be handled analogously to the heartbeat). This movement takes place 
relatively quickly compared with the production of projection photographs with the X-ray 
apparatus 1. This leads to the situation that in a recorded sequence Ai, ... An, ... An, the 
projection photographs originate from different heartbeat phases, and in fact as a rule in an 
unordered sequence. Starting from this circumstance, with the proposed method, and taking 
30 account of the movement through the heartbeat, the spatial position of the stent 5 is to be 
reconstructed as accurately as possible. 

In order to achieve the objective mentioned above, in parallel with the X-ray 
projections A n a parameter is recorded which indicates the heartbeat phase. This can be for 
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example the electrical signal En of an electrocardiogram, which is recorded with the 
electrocardiograph device 2. 

As a result of the preparatory recording procedure represented in block o of 
Figure 4a, we thus have a sequence (Ai, Ei), ... (A n , En), ... (An, En) of X-ray pictures A n 
5 together with the respective corresponding ECG phase En which is stored in the memory 3 of 
a data processing unit 4. 

In the first processing step (block a) of the method, on all the projection 
photographs A n the images Rn, Q n of feature points R, Q are then segmented. The feature 
points are features that are easily visible on the photographs, such as for example markers (on 

10 a stent 5, a balloon or a guide wire), or anatomical features such as bifurcations of vessels. As 
a result of the segmentation, for each photograph A n we know the image co-ordinates (£, 
ORn, (£» Qqh related to this photograph. Depending on the nature of the feature point, 
segmentation can take place fully automatically (for example in the case of markers) or semi- 
automatically (for example in the case of anatomical structure), i.e. with interactive support 

15 by a user. 

In block b of the method, classification of the available projections An into 
classes Ki, ... Kp, ... Km takes place: these correspond respectively to a phase (or, to be more 
precise, a phase interval) Ej C1 , ... Ep CI , ... E M C1 of the cyclical self-movement. The division of 
a heartbeat phase into phases Ep C1 (of equal or different lengths) is given here. The classes 

20 thus collect the X-ray pictures A n together into clusters that originate from (roughly) the same 
heartbeat phase. The classes can be defined formally e.g. as index quantities according to 
K p - {n|EneEp C1 }. 

In block c, for each of the given heartbeat phases Ep C1 , the three-dimensional 
"real" co-ordinates (x,y,z)Rp, (x,y,z)op of the feature points R and Q are calculated from 

25 respectively (at least) two projection photographs A n i, An2, wherein said photographs should 
originate from the same heartbeat phase Ep C1 (i.e. nl, n2 e K p ). For this calculation, we refer 
back to the image co-ordinates (£, QRn, (£, Q Qn determined in block a, as well as to the 
projection centers F n i, Fn2 that are known from the calibration of the equipment. The 
necessary calculation methods are, moreover, sufficiently well known from the prior art. 

30 In the next block d, the field of movement for each feature point R, Q is 

calculated in relation to the heartbeat. Here, the spatial co-ordinates (x,y,z)Rp, (x,y,z)op, 
(x,y,z)Rm, (x,y,z)Q m of the feature points known from block c are used in different heartbeat 
phases p, m, in order to determine for each feature point R, Q a three-dimensional field of 
movement vectors or displacement vectors S R p _m, S Q P-m . The displacement vector S Q P-m here 
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describes for example the displacement of the feature point Q with the co-ordinates (x,y,z)op 
in the heartbeat phase p in the transition to the co-ordinates (x,y,z)Q m in the heartbeat phase 
m. 

Based on the displacement vectors S R p _ m , S Q p _ m » furthermore, in block d 
5 transformations E p _ m are determined for the movement of the whole stent* 5 or its 

surroundings. In the simplest case, a rigid translation can be assumed, for the determination 
of which the displacement vector of a single feature point is sufficient. Usually however 
more realistic movement models are required, in which at least two feature points R, Q are 
used for determining a translation, rotation and/or dilation. Using several feature points, it is 

10 also possible to determine more complex, for example affine, transformations. 

In the last block e, based on the three-dimensional movement model of the 
object 5 (block d), a reconstruction - which is compensated in respect of the three- 
dimensional movement - takes place of the whole stent or region of interest, using projection 
photographs from different heartbeat phases. In the reconstruction of each image volume 

15 element (voxel) which is depicted on two projection photographs A n and Aj from different 
heartbeat phases pi, p2, with regard to the corresponding object point it is taken into account 
that in the heartbeat phases of the photographs it lay in different positions r p i and Tj>2. The 
aforementioned positions are linked here by the known transformation Spi^- With the aid of 
this transformation, one can thus reconstruct the region of interest and in particular the stent 5 

20 by using the information from all projection photographs with a high level of accuracy;; since 
the influence of the heartbeat movement can be compensated by the transformations. . 



